1. Technical Field
The invention relates to data processing. More particularly, the invention relates to a method and apparatus for pipelining the transformation of information between heterogeneous sets of data sources.
2. Description of the Prior Art
Enterprise application integration is a new market focused on the exchange of data between legacy and packaged applications. The traditional way of building interfaces between applications has been to extract data from the source application, stage it in some mass storage, then transform it according to some transformation rule, stage the transformed data in mass storage and deliver it to the target application. There are three problems with this source-oriented approach:
The amount of mass storage necessary to stage the data;
The overhead in reading and writing the data to mass storage multiple times; and
The serial execution of the extraction, transformation, and delivery does not take advantage of multiprocessor technology.
Staging is necessary because the target data are often the result of transforming individual data attributes that are scattered all over the source. It would be advantageous to provide a method and apparatus that addresses the problems attendant with the transformation of information between heterogeneous sets of data sources.
The invention provides a method and apparatus for pipelining the transformation of information between heterogeneous sets of data sources. The invention avoids intermediate data staging by constructing metadata that allow the acquisition of data in an order that matches the order in which the target data must be delivered.
In a further feature of the invention, the acquisition, transformation, and delivery stages of an interface between two applications are pipelined as three processes that execute simultaneously on a continuous flow of data.
Pipelining is a well-known technology that has been in use in operating systems, such as UNIX pipes and more recently by Torrent systems, to optimize the usage of multiprocessor machines. The invention applies pipelining technology to data transformation between heterogeneous data sources.